An Empirical Study of Scenarios Gained and Lost in Architecture Evaluation Meetings

An important element in scenario-based architecture evaluation is the development of scenarios by holding meetings of stakeholders. As the team meeting is an expensive activity, studying the effectiveness of meetings is an important research question. In this paper, we report the findings from analyzing the data collected in a controlled experiment aimed at empirically studying the effectiveness of scenario development meetings in terms of gained and lost scenarios. A secondary researched issue was whether or not a top-down technique for eliciting scenarios can improve the performance of a team meeting compared to a bottom-up technique. Findings from data analysis question the effectiveness of holding meetings for developing scenarios since more important scenarios were lost than gained in these meetings. Data results also provide empirical support to our assertion that top-down scenario development technique is better than the bottom-up technique.